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AUGMENTED AND VIRTUAL REALITY GUIDED INSTRUMENT 
POSITIONING USING ALONG -THE -LINE -OF -SIGHT ALIGNMENT 

5 BACKGROUND 

1. Technical Field 

The present invention generally relates to 
augmented reality and, in particular, to a method for 

10 augm^nt^d reality guided instrument positioning using 
along-the-line-of -sight alignment . 

2. Background Description 

Augmented reality, also commonly referred to as 
15 augmented vision or augmented reality vision, augments 

an observer's view of the real world by superimposing 
computer generated graphical information. This 
information may be as simple a.s a. text label attached 
to some object in the scene, or as complex as a 3D 

20 mod^l of a patient's brain derived from an MR I scan and 
aligned to the real view of the person's head. 

The observer may observe a real scene directly 
with his or her eyes, with the additional graphical 

information being blended therewith via a semi- 
25 transparent display located between the observer and 

the real scene. Su.ch. a display device can be, for 
example, a see -through head mounted display. 




The display can also be opaque, like a computer 
screen or a non- see -through head mounted display. Such 
a display then presents to the observer the complete 

augmented view, i.e., a combination of the real-world 
5 view and the graphics overlay. A video camera takes 

the place of the real -world observer to capture the 

real world-view. For stereo vision, two cameras are 
required. A computer is used to combine the live video 
with the graphics augmentation. 
10 The graphics have to be positioned, oriented, and 

scaled, or even rendered in a perspective fashion for 
correct alignment with the real-world view. It is 

desirable to n anchor" the graphics to a real-world 
object. To do this, the position and orientation of 
15 the camera with respect to the object, as well as the 

orientation of the object, must be known. That is, the 
relationship between two coordinate systems, one 

Corresponding to the camera and the other corresponding 
to the object, must be known. 

20 Tracking denotes the process of keeping track of 

the preceding relationship. Commercial tracking 
systems are available that are based on optical, 

mechanical, magnetic, inertial, and ultrasound 

measurement principles . 

25 FIG. 1 is a block diagram illustrating an 

augmented reality system 100 wherein video images of 




the real world are combined with computer generated 
graphics, according to the prior art. The system 100 
includes: a video camera 110; external trackers 112; 
2D/3D graphics module 114; an image processing module 

5 116; a pose calculation module 118; a graphics 

rendering module 120; a video and graphics overlay 
module 122; and a display 124. As is known, a 3D 
perception may be achieved through the use of two 

cameras and a stereo display. 

10 The operation of the elements typically employed 

in an augmented reality system as well as the 
calibration required of such a system is described by: 
Ahlers et al . , in u Calibration Requirements and 
Procedures for a Monitor-based Augmented Reality 

15 System" , IEEE Transactions on Visualization and 

Computer Graphics, 1(3): 255-273, 1395; Navafr et al., 

in "Single Point Active Alignment Method (SPAAM) for 
Calibrating an Optical See-through Head. Mounted. 
Display", Proc . of the IEEE International Symposium on 

20 Augmented Reality, ISAR '00, Munich, Germany, Oct. 

2000; and. Saner: et al . , "Augmented Workspace : Designing 
an AR Testbed" , Proc. of the IEEE International 
Symposium on Augmented Reality, ISAR '00, Munich, 
Germany, Oct. 2 000. 

25 Augmented reality visualization can guide a user: 

in manual mechanical tasks. For machine repair and 



maintenance scenarios, it has been suggested to augment 

the view with graphical pointers that show, e.g., which 

button to press or which screw to turn. Augmented 
reality visualization is also being suggested for 
5 medical applications where, e.g., biopsy needles have 

to be inserted into a target tumor without harming 

nearby nerves or where screws have to be inserted into 

bones at a precise location and in a precise direction. 

A description of one of the main problems with 
10 augmented reality visualization, i.e., occlusion, will 

now be given. As noted above, augmented Reality 

visualization places virtual objects (computer 

generated graphics) into real scenes. The tracking of 
the vantage point, from which the real scene is viewed., 
15 with respect to a world coordinate system anchored at 
real world objects, allows the virtual objects to 

appear at desired locations in this world coordinate 
system. However, a correct visual interaction between 

real and virtual objects generally requires 3D 
20 information about the real objects. Disadvantageously , 
this 3D information is usually not available and, thus, 

the virtual objects are simply superimposed onto the 

image of the real scene. Accordingly, real objects can 
be hidden by virtual objects, although virtual objects 
25 cannot be hidden by real objects. 



To illustrate the problem of occlusion , consider 
the following example: a simple virtual object such as 
a cube is superimposed onto a real scene, with all the 

real objects being farther away from the observer than 

5 the virtual cube. If this augmented scene is shown to 

the observer in stereo, the observer can easily 

perceive the correct 3D position of the cube. Now the 
observer inserts his hand in the scene so that his hand 

is in the foreground and occludes the position where 

10 the virtual cube is supposed to be. Since the system 

has no knowledge of the hand/ the system still shows 

the virtual cube superimposed onto the scene image. 

The inability of the (real) hand to occlude the virtual 

cube confuses the 3D perception o£ the observer. The 
15 observer receives conflicting depth cues: the stereo 

depth cues tell the observer that the cube is behind 

the hand, the occlusion cue tells the observer that the 

cube must be in front of the hand (because otherwise it 
would be hidden by the hand) . 
20 Accordingly, it would be desirable and highly 

advantageous to have a method for augmented reality 

guided instrument positioning which is not adversely 

affected by occlusion. 



SUMMARY OF THE INVENTION 

The problems stated above, as well as other 

related problems of the prior art, are solved by the 

present invention, methods for augmented and virtual 
5 reality guided instrument positioning using along-the- 

line- of -sight alignment . 

According to an aspect of the invention, there is 

provided a method for augmented reality guided 
instrument positioning. A viewpoint is established, 

10 from which a line of sight to a point on a target 

defines a path for an instrument to follow during a 
positioning of the instrument to the point on the 
target. The instrument is aligned along the line of 
sight to the point on the target . 

15 According to another aspect of the invention, 

there is provided a method for augmented reality guided 
instrument positioning. At least one graphics path 

guide is rendered for indicating a path for an 

instrument to follow to a target. At least one axis 
20 marker is identified on the instrument that marks an 

axis of the instrument. The axis is disposed from a 
front portion to a back portion of the instrument. The 

front portion corresponds to the point on the target. 
The at least one axis marker is aligned with respect to 

25 the at least one graphics guide to align the instrument 

to the path. 
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According to yet another aspect of the invention, 
there is provided a method, for virtual reality guided 
instrument positioning. A point on an actual target 
and an actual path to reach the point on the actual 

5 target are defined, h pose of an actual instrument is 

tracked with respect to a pose of the actual target. A 
graphical representation of the actual instrument and. 

the actual target point is rendered to obtain a virtual 
instrument and a virtual target point, respectively. 

10 The graphical representation is rendered with respect 

to a virtual viewpoint from which a virtual line of 
sight to the virtual target point coincides with a 

virtual path for the virtual instrument to follow 

during a positioning of the actual instrument to the 
15 point on the actual target. The virtual path 

corresponds to the actual path. The virtual instrument 

is aligned along the virtual line of sight to the 
virtual target point to align the actual instrument 
along the actual path. 
20 These and other aspects, features and advantages 

o£ the present invention will become apparent from the 

following detailed description of preferred 

embodiments, which is to be read in connection with the 

accompanying drawings. 

25 



BRIEF DESCRIPTION OF THE DRAWINGS 

PIG. 1 is a block diagram illustrating an 
augmented reality system 100 wherein video images of 
the real world are combined with computer generated 
graphics, according to trie prior art; 

FIG. 2 is a block diagram of an augmented reality 
guided instrument positioning system, according to an 
illustrative embodiment of the present invention; 

FIG. 3 is a diagram illustrating a graphics guide 
3 00 for augmented reality guided positioning of a 

needle, according to an illustrative embodiment of the 
present invention; 

FIG. 4 is a diagram illustrating another graphics 

guide 400 for augmented reality guided positioning of a 
needle, according to another illustrative embodiment of 
the present invention; 

PIG. 5 is a diagram illustrating yet another 
graphics guide 500 for augmented reality guided 

positioning of an instrument, according to yet another 

illustrative embodiment of the present invention; 

FIG. 6 is a diagram illustrating still another 
graphics guide 600 for augmented reality guided 

positioning of a needle, according to still another 

illustrative embodiment of the present invention; 

FIG. 7 is a flow diagram illustrating a method for 
augmented reality guided instrument positioning using 



graphics guides , according to an illustrative 
embodiment of the present invention; 

FIG. 8 is a diagram illustrating a modulated 
graphics guide 800 for augmented reality guided 

positioning of a needle, according to an illustrative 
embodiment of the present invention; 

PIG. 9 is a diagram illustrating another modulated 
graphics guide 900 for augmented reality guided 
positioning of a needle, according to another 
illustrative embodiment of the present invention; 

FIG. 10 is a diagram illustrating yet another 
modulated graphics guide 1000 for augmented reality 
guided positioning of a needle, according to yet 
another illustrative embodiment of the present 
invention; 

FIG. 11A is a flow diagram illustrating a method 

for augmented reality guided instrument positioning 

using modulated graphics guides, according to an 

illustrative embodiment of the present invention; 

FIG. 11B is a flow diagram illustrating another 
method for augmented reality guided instrument 

positioning using modulated graphics guides, according 

to another illustrative embodiment of the present 
invention; 

FIG. 11C is a flow diagram illustrating yet 

another method for augmented reality guided instrument 




positioning using modulated graphics guides, according 
to yet another illustrative embodiment of the present 
invention; 

FIG. 12 is a simplified diagram illustrating a 
5 needle 12 0 before and after being placed in alignment 

with an observer's line of sight, according to an 
illustrative embodiment of the present invention; 

PIG. 13 is a diagram illustrating a conventional 

instrument 1300 having a structure that renders the 

10 instrument 1300 "not alignable" ; 

FIG. 14 is a diagram illustrating various views of 
the instrument 1300 of FIG. 13, with a marker 1430 
located thereon according to an illustrative embodiment 
of the present invention; 

15 FIG. 15 is a diagram illustrating graphics path 

guides 1510, 1520, and 1530 for indicating a path 1550 
for an instrument to follow to a target 1560, according 

to an illustrative embodiment of the present invention; 

FIG. 16 is a diagram illustrating side and rear 
20 views of an instrument 16 0 0 having an axis marker 1620, 

according to an illustrative embodiment of the present 
invention; 

FIG. 17 is a diagram illustrating graphics path 
markers 1710, 1720, and 1730 and an axis marker 1750 
25 for an instrument 1700, according to an illustrative 
embodiment of the present invention; 



FIG. 18A is a flow diagram illustrating a method 
for augmented reality guided positioning of an 
instrument using a bull's eye alignment approach, 
according to an illustrative embodiment of the present 
invention ; 

FIG. 18B is a flow diagram illustrating another 
method for augmented reality guided positioning of an 
instrument using a bull's eye alignment approach, 
according to another illustrative embodiment of the 

present invention; 

FIG. 19 is a diagram illustrating a graphics 
instrument guide (virtual instrument) 1910 and 192 0 and 
a graphics target guide (virtual target) 1940 for 
virtual line-of -sight guided positioning of an 

instrument according to an illustrative embodiment of 

the present invention; 

FIG. 20 is a diagram illustrating another graphics 
instrument guide (virtual instrument) 2 010 and 2 02 0 and 
another graphics target guide (virtual target) 2 04 0 for 
virtual line-of -sight guided positioning of an 
instrument according to another illustrative embodiment 

of the present invention; 

FIG. 21 is a diagram illustrating graphic depth 
markers for graphically representing instrument depth 

in virtual line-of-sight guided positioning of an 



instrument, according to an illustrative embodiment of 

the present invention. 

FIG. 22 is a diagram illustrating graphics depth 
marks and graphics path marks for virtual 1 ine - o£ - Signt 
5 guided positioning of an instrument, according to an 
illustrative embodiment of the present invention; 

PIG. 23 is a flow diagram illustrating a method 
for virtual line-of -sight guided positioning of an 
instrument, according to an illustrative embodiment of 

10 the present invention; 

FIG. 24 is a diagram illustrating a graphics depth 
marker 24 00 for augmented reality guided positioning of 

an instrument, according to an illustrative embodiment 
of the present invention; 
15 FIG. 25 is a diagram illustrating a graphics depth 

marker 2500 added to a graphics path guide 2510 for 
augmented reality guided positioning of an instrument, 

according to an illustrative embodiment of the present 

invention; 

20 FIG. 26 is a diagram illustrating instrument 

markers used as depth markers for augmented reality 
guided positioning of an instrument, according to an 
illustrative embodiment of the present invention; and 
FIG. 27 is a flow diagram illustrating a method 
25 for indicating instrument depth for augmented reality 



cp_iid.ed. instrument positioning, according to an 
illustrative embodiment of the present invention. 

detailed DESCRIPTION OP PREFERRED EMBODIMENTS 

5 The present invention is directed to methods for 

augmented and virtual reality guided instrument 
positioning using along-the-line-of -sight alignment. 

It is to be understood that the present invention 
may be implemented in various forms of hardware, 

10 software, firmware , special purpose processors, or a 

combination thereof. Preferably, the present invention 
is implemented as a combination of hardware and 
software. Moreover, the software is preferably 
implemented as an application program tangibly embodied 
15 on a program storage device. The application program 

may be uploaded to, and executed by, a machine 
comprising any suitable architecture. Preferably, the 
machine is implemented on a computer platform having 
hardware such as one or more central processing units 

20 (CPU) , a random access memory (RAM) , and input/output 

(I/O) interface (s) . The computer platform also 

includes an operating system and microinstruction code. 
The various processes and functions described herein 
may either be part of the microinstruction code or part 
25 of the application program (or a combination thereof) 
which is executed via the operating system. In 




addition, various other peripheral devices may be 

connected to the computer platform such as an 
additional data storage device and a printing device. 

It is to toe further -understood that, because some 
of the constituent system components and method steps 
depicted in the accompanying Figures are preferably 
implemented in software, the actual connections between 
the system components (or the process steps) may differ 
depending upon the manner in which the present 

invention is programmed. Given the teachings herein, 

one of ordinary skill in the related art will be able 
to contemplate these and similar implementations or 
configurations of the present invention. 

pig . 2 is a block diagram o£ an augmented reality 

guided instrument positioning system 200, according to 
an illustrative embodiment of the present invention. 

The system 200 includes! two video cameras 210a-b; a 

tracking module 212; a pre-processing 2D/3D graphics 
module 250; a path determining device 260; a post- 
processing 2D/3D graphics module 270; a graphics guide 

generator 280; an image processing module 216; a pose 
calculation module 218; a graphics rendering module 
22 0; a video and graphics overlay module 222; and a 
stereo display 224. The two video cameras 210a-b and 
the stereo display 224 are employed to obtain 3D 
perception. Of course, other arrangements may be used 




(e.g., single video camera and non-stereo display) 

which maintain the spirit and scope of the present 

invention. In the illustrative embodiment of FIG. 2, 
the path determining module 2 60 and the graphics guide 

generator 280 are stand alone elements o£ the system 

200. However, as readily appreciated by one of 
ordinary skill in the related, these elements may be 

integrated into Other elements of the system 200 (e.g., 
the graphics rendering module 220) . Given the 
teachings of the present invention provided herein, one 

of ordinary skill in the related art will contemplate 

these and various other configurations of the elements 
of FIG. 2, while maintaining the spirit and scope of 
the present invention. 

The following description of the elements of FIG. 

2 will be given with respect to augmented reality 

guided instrument applications. The two video cameras 
210a-b provide image related data corresponding to the 
real world to the system 200. The tracking module 212 
provides data corresponding to the relationship between 

a user coordinate system (i.e., with respect to the 

vantage point of the user or the vantage point of the 
cameras that replace the eyes of the user) , a 

coordinate system of the object (e.g., patient) to 

which the instrument is being applied and, optionally, 
an instrument coordinate system. The data provided by 
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the tracking module 212 allows for the relationship (s) 

between the preceding coordinate systems to be known or 

finally calculated in the pose calculation module 218. 

The data provided through the pre-processing 2D/3D 

graphics module 250 may be, e.g., a Computer 
Tomographic (CT) scan or Magnetic Resonance (MR) image, 

preprocessed for a specific instrument POSltlOning 
application. The data provided through the pre- 
processing 2D/3D graphics mo dule 250 may also include 
geometrical information about the instrument. From 

information output from the pre-processing 2D/3D 

graphics module 250 to the path determining module 260, 
one or more potential paths for the instrument to 
travel and/or one or more potential target locations 

for the ins tmment to contact/penetrate may be 

determined. The path/target data as well as some data 
passed through the path determining module 2 SO £roft\ the 
pre-processing 2D/3D graphics module 250 is input to 
the post -processing module 2 70 which provides the data 

to the graphics guide generator 280 for use in 

generating graphics guides. It is to be appreciated 
that data which specifies one or more preferred types 

of graphics guides to be used during the positioning of 

the instrument may be input to the pre-processing 2D/3D 

graphics module 25 0 for ultimate use by the graphiC9 
guide generator 280 in selecting a type(s) of graphics 




guide to generate . Some illustrative types of graphics 
guides are shown in FIGs. 3 through 6. 

The image processing module 216 processes the 
image data provided from the two video cameras 210a-b. 
5 Processed image data from the image processing module 
216 is input to the pose calculation module 218 along 

with the data provided from the tracking module 212. 

Data from the graphics guide generator 2 80 
corresponding to one or more graphics guides is input 

10 to the graphics rendering module 22 0 along with the 
data output from the pose calculation module 218 so 
that the graphics can be rendered according to the 
camera pose, to be registered with respect to the 
object (to which the instrument is to be applied) seen 

15 in the video images. Rendered data is provided to the 
video and graphics overlay module 222 for subsequent 

display on the stereo display 224, 

The operations and capabilities of the elements of 

system 200 are further described in detail With T^SpSCt 
20 to FIGs. 3-7. For the sake of brevity, additional 
conventional operations performed by any of the 
elements of system 200 have essentially been omitted. 
For additional details on these conventional 
Operations, please see, e.g.: Ahlers et al., in 
25 "Calibration Requirements and Procedures for a Monitor- 

based Augmented Reality System", IEEE Transactions on 
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visualization and Computer Graphics; 1(3): 255-273, 

1995; Navab et al . , in "Single Point Active Alignment 
Method (SPAAM) for- Calibrating an Optical See - thrOUOfh 
Head Mounted Display", Proc. of the IEEE International 
5 Symposium on Augmented Reality, ISAR "00, Munich, 

Germany, Oct. 2000; and Sauer et al., "Augmented 

Workspace: Designing an AR Testbed" , Proc. of the IEEE 
International Symposium on Augmented Reality, isar ' oo, 

Munich, Germany, Oct. 2000. 

10 The present invention is directed to augmented 

reality guided instrument positioning with guiding 
graphics. The invention may be employed in the case 
when an instrument has to be positioned with a fairly 
high accuracy, and where a target or a target and a 

IS path can be pre-selected and can be shown to the user 

as graphical guides in an augmented reality view. The 
guiding graphics mark the target. The guiding graphics 
may also possibly mark the final instrument position 

and/or on entation. Further, the guiding graphics may 

20 possibly mark the path to the final instrument pose. 

The user then aligns the instrument with the graphics 
guides to perform his or her task. 

It is to be appreciated that the occlusion problem 

presents a significant obstacle to using augmented 

25 reality for instrument guidance. The real instrument 

has to be brought close to the location of the virtual 



guides, and only if one can clearly perceive their 
relative 3D locations can one perform the alignment 
reliably . 

Consider the example of a graphics guide for 

positioning a needle. A cylinder shaped guide can be 

used to mark the desired needle position, the axis of 
which lies on the desired needle path, and an end of 

which can mark the desired position of the needle tip 

if this position is on the surface or close to the 
surface. It may also mark the needle entry point into 
the object. If this guiding cylinder is rendered solid 

(or semi-transparent) , the guiding cylinder will 

OCClude the needle in a way that is independent of 
whether the needle is closer to or farther from the 
observer's vantage point. This is the occlusion 
problem: the wrong occlusion cues disturb the correct 
3D perception. 

To minimise the effect of the occlusion problem, 

the invention advantageously constructs guides 
according to the following principle: when the needle 
gets close to its desired position, guide and needle 
should not overlap significantly, if at all, in the 
augmented view. Rather than marking the correct 
position directly, the guide is framing the correct 
position. The path where the needle is supposed to be 
placed is "empty" in the graphics guide. When the 



needle is placed in the correct position, guiding 

graphics and needle appear side by side, avoiding 

occlusion that disturbs depth perception. 

FIG. 3 is a diagram illustrating a graphics guide 

5 300 for augmented reality guided positioning of a 

needle, according to an illustrative embodiment o£ the 
present invention. The graphics guide 300 includes a 
rectangle 310 with a (transparent) window 320 that 

marks the needle path 350. It is to be appreciated 
10 that the graphics guide 300 could be rendered solid, 

half-transparent, textured , outlined, and so forth. 

The important feature in any of the preceding 
configurations is the transparent window 32 0 at the 
desired location of the needle. It is to be appreciated 
15 that while a rectangle is used in the embodiment of 

FIG. 3, any shape can be used to implement the graphic 
guides of the invention while maintaining the spirit 

and scope thereof. For example, polygons/ circles, and 

so forth may be used a graphics guides according to the 
20 invention. 

FIG, 4 is a diagram illustrating another graphics 
guide 400 for augmented reality guided positioning of a 
needle, according to another illustrative embodiment of 

the present invention. The graphics guide 400 is 

25 derived from the cylindrical guide described above and 

includes a cylinder 410 with a (transparent) window 420 
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that marks the needle path 4 50. The rendering of only 

the surface of the cylinder 410 and the cutting out of 

the transparent window 420 along the axis through the 
cylinder 410 eliminates occlusion between the graphics 

guide 400 and the needle once the needle is aligned 

with the cylinder's axis. 

FIG. 5 is a diagram illustrating a graphics guide 
50 0 for- augmented, reality g-u.id.ed positioning o£ an 
instrument, according to an illustrative embodiment of 
the present invention. In this example, the graphics 
guide marks a target 502 onto which the tip of an 

instrument should be placed (the instrument's 
orientation being arbitrary) , according to yet another 
illustrative embodiment of the invention, e.g., a 
target on the object's surface, or an entry point of 
the needle. The graphics guide 500 includes a circle 
510 having an opening 520 so as to resemble the letter 
C. The target 502 is framed by the circle 510 
(preferably, but not necessarily, centered) , and the 
outline of the circle 510 is broken (i.e., the opening 
52 0) on the side where the tip of the needle is 
expected to enter the circle 510. For example, the 
outline of the circle 510 is broken on the lower right 
side when the user is right-handed, or on the lower 
left side when the user is left-handed. Thus, the 
opening may be used to specify a potential direction of 



approach or path 550 for the needle. Advantageously, 
the tip of the instrument can reach the target 5 02 
without getting occluded by the graphics guide 500. 

FIG. 6 is a diagram illustrating still another 
graphics guide 600 for augmented reality guided 
positioning of a needle, according to still another 
illustrative embodiment of the present invention. The 
graphics guide 600 includes a pattern of lines. The 
graphics guide 600 is preferably centered on the target 
602 or, at the least, surrounds the target. The lines 
may be equally spaced (as shown) to allow an elongated 
member of pass through any two consecutive lines. 
Alternatively, at least two consecutive lines may be 
spaced differently from the remaining lines to frame a 
path 650 of the instrument to the target 602. It is to 
be appreciated that in the latter case the non- 
symmetrical spacing enables an observer to readily 
determine the path 650 to the target location. 

A description of vantage point dependent graphics 
guides will now be given according to an illustrative 
embodiment of the present invention. It is to be 
appreciated that if the observer can assume a greater 
range of positions, it is advantageous to change the 
orientation of the graphics guides according to the 
observer's vantage point. For example, in the case of 
the windowed rectangle of FIG. 2, the windowed 




rectangle can be rotated around the axis of the needle 
path so that it keeps facing the user. In the case of 
the windowed cylinder of FIG. 3, the slit can be 
rotated so that the user keeps looking through the 
5 slit. The information required to a.dj-u.st the graphics 

guides according to the user's vantage point is readily 
available as the vantage point is already being tracked 

for correct registration between virtual objects and 
the real scene. 

10 It is to be appreciated that while a needle has 

been used as the instrument to be positioned in the 
examples of FIGs . 3-6, any type of instrument may be 

employed. Far example, with respect tc the graphic 

guides of FIGs. 3-6, the instrument may be any 
15 instrument that includes an elongated member such as, 

for example, a scalpel, a drill, and so forth. Given 

the teachings of the invention provided herein, one of 
ordinary skill in the related art will contemplate 
these and various other instruments to which the 

20 present invention may be applied. 

FIG. 7 is a flow diagram illustrating a method for 
augmented reality guided instrument positioning using 
graphics guides, according to an illustrative 
embodiment of the present invention. In the 

25 illustrative embodiment of FIG. 7, the instrument 

includes an elongated member with a tip. However, 



other instruments may be substituted therefore, while 
maintaining the spirit and. scope of the invention. 

A potential path to be traversed during a 
potential positioning of the instrument is identified 
a.s a predetermined path (step 710) . A graphics guide 

for positioning the instrument is determined such that 
the graphics guide provides a substantially 

unobstructed view of the instrument during the 

positioning (step 720) . For example, in the case of a 
drill; only the bit or the portion thereof that 
contacts a patient need be unobstructed; it is 
generally inconsequential if the drill housing is 
obstructed. That is, the invention provides for- some 
or all of the contact /insert ion medium of the 
instrument to be unobstructed, while the handle, 

housing, or other patient non-contact/non-insertion 

portion may or may not be obstructed. It is to be 
appreciated that, in some circumstances, there may be 

some occlusion of the contact/insertion medium during 

rough (initial) positioning of an instrument. For 
sample, in the case or the cylinder o£ PIG. 4, a user 
(e.g., physician) may enter the needle into the 
cylinder through the sidewall of the cylinder, thereby 
temporarily occluding the needle; however, once the 

needle is within the cylinder, there is no occlusion of 
the needle by the cylinder (due to the window) . 



The graphics guide is constructed to frame the 
predetermined path (step 730) . The predetermined path 

is framed with a window for steps 730a-b, an opening 

for step 730c, and a pattern of lines for step 730d. 
it is to be appreciated that the window may be 

transparent or semi-transparent. In the latter ca&e, a 

shading scheme may be used such that it is clear at the 
centerl ine of the window but becomes increasingly 
shaded in directions perpendicular to the centerline of 

the window (with the centerline of the window aligned 
with trie path) . 

With respect to step 730a, a polygon is 
constructed that includes two opposing ends and the 
window disposed from one of the two opposing ends to 
another one of the two opposing ends. 

With respect to step 730b, a partial cylinder is 
constructed that has a first end and a second end and 
the window disposed from the first end to the second 

end along a length-wise axis o£ the partial cylinder. 

With respect to step 730c, a circle is constructed 
that has an opening such that the circle resembles the 
letter C, and the opening is larger than a width of the 
elongated member. 

With respect to step 730d, a pattern of lines is 

constructed such that at least two consecutive lines o£ 



the pattern of lines are spaced to allow the elongated 

member to pass through. 

The observer and/or instrument are tracked (step 
735) , so that their pose with respect to the 
object (patient) /target can be determined. The graphics 
guide is rendered (e.g., solid, semi-transparent, 
textured, outlines, or a combination thereof) in 
alignment with the predetermined path to be traversed 

during the positioning of the instrument, (step 740). 

Step 74 0 includes orienting the graphics guide with 
respect to the observer (step 750) . Such orientation 
is performed according to steps 750a, 750b, 750c, or 
650d depending upon whether step 730a, 730b, 730c, or 
730d was previously performed, respectively. Step 740 
may optionally include rendering a representation of 
the instrument during the positioning to reveal an 
augmented reality spatial representation of the 
instrument with respect to the graphics guide. 

With respect to step 750a, the polygon is oriented 

so that a centerline of the window is aligned with the 
predetermined path. 

With respect to step 750b, the partial cylinder is 
oriented so as to have the length-wise axis of the 
cylinder aligned with the predetermined path. 

With respect to step 750cl, the circle is oriented 
so as to substantially enclose the target location and 



the opening is one of to the right with respect to the 
Observer When the observer is right-handed or to the 
left with respect to the observer when the observer is 
left-handed. 

With respect to step 7S0c2, the circle is oriented 
such that the circle substantially encloses the target 
location and the opening encloses an axis of the at 
least one path. 

With respect to step 750d, the pattern of lines is 
oriented to surround a target location of the elongated 

member, with the at least two consecutive lines framing 

the predetermined path. It is to be appreciated that 
m Some embodiments or the invention the pattern of 
lines may simply be oriented to frame the target 
location irrespective of the path, when any two 
consecutive lines are spaced to allow the elongated 
member to pass therethrough. In other embodiments of 
the invention, the at least two lines that frame the 
path may be made longer than the remaining lines or 
have a different spacing than the remaining lines to 

indicate that the instrument is to travel therebetween. 

Steps 750a or 750b may include the step of 
dynamically adjusting (or re - ad j ust ing) the orientation 
of the windowed guide relative to the vantage point of 
the observer (step 760ab) . 




Steps 750cl or 750c2 may include the step of 

dynamically adjusting (or re-adjusting) the orientation 

of the opening relative to the vantage point of the 
observer (step 760c) . 

Step 750d may include the step of dynamically 

adjusting (or re - ad j us ting) the orientation of the 

pattern o£ lines relative to the vantage point of the 
observer . 

A description of modulated graphics guides for 
augmented reality guided instrument positioning will 
now be given according to an illustrative embodiment of 
the present invention. 

To minimize the effect of the occlusion problem, 
the appearance of a graphics guide is modulated. 
Modulation means that the guides are not constructed as 
extended uniform objects. Rather, the transparency of 
the graphics guides (or portions thereof) are varied. 
The variation can be implemented spatially, temporally, 
or a combination thereof. For example, the variation 
can be in time with the guiding graphics (or portions 
thereof) fading in and out in the augmented image to 
repetitively permit an unobstructed view of the 
instrument at pre-defined intervals. The variation can 

also be in space with the graphics guides having high 

transparency regions that permit an unobstructed view 
of some or all of the instrument. The variation can 



also be both in space and in time, for example with 

high transparency regions moving across the graphics 

guides. Other possibilities for space and time 
variation are readily contemplated by one of ordinary 
skill in the art while maintaining the spirit and scop 
of the present invention. 

Consider, for example, the case wherein the tip o 

the instrument needs to be placed onto a target with 
arbitrary orientation of the instrument, and a simple 
disc Or ball shaped graphics object is used to mark th 
target. When the augmented view shows the marker 
overlapping with (and occluding) the tip of the 

instrument while the tip of the instrument is actuallv 

still in front of the target location, this wrong 
occlusion disturbs the correct 3D perception. A simple 

temporal modulation of the marker (i.e. varying its 

transparency so that it fades in and out) will reduce 
the negative impact of wrong occlusion. 

FIG. 8 is a diagram illustrating a modulated 
graphics guide 800 for augmented reality guided 
positioning of a needle, according to an illustrative 
embodiment of the present invention. The modulated 
graphics guide 800 includes a thick discontinuous line 
having a plurality of closed segments 82 0 and a 
plurality of open segments 830. The modulated 
graphics guide 800 is aligned with the desired path 850 



for the needle to travel to the target. It is to be 
appreciated that the modulated graphics guide could be 

rendered solid, half -transparent , textured, outlined, 
and so forth. The important feature of the graphics 

guide of FIG. 8 is the modulation (here spatial) that 

permits a substantially unobstructed view of the needle 
through the open segments 83 0. It is to be further 

appreciated that the transition between the space 

segments (open segments 83 0) and the line segments 
(closed segments 820) needs not t>e a.t>rri_Lpt as shown but 
could be a smooth transparency transition over an 
extended distance. Additional temporal modulation could 

make the line segments (closed segments 820) fade in 

and out, could make the spaces (open segments 83 0) fade 
In and OUt, COUld makG the spaces move along the line, 
switch transparencies between different regions of the 
line, and so forth. In a purely temporal modulation, 

the whole line (without open segments 820) would fade 

in and out . 

fig. 9 is a diagram illustrating another modulated 

graphics guide 900 for augmented reality guided 

positioning of a needle, according to another 

illustrative embodiment of the present invention. The 

modulated graphics guide 90 0 includes a discontinuous 
cylinder having a plurality of closed Segments 920 and 
a plurality of open segments 930. An axis of the 
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cylinder is preferably centered or, at the least 



surrounds a desired path 950. The plurality of open 



eliminate occlusion between the guide and the needle in 



Figure 8, different renderings of the cylinder segments 
(closed segments 820, open segments 830) are possible, 
as well as different ways of combining temporal and 



spatial modulation. In a purely temporal modulation, 



the whole cylinder would fade in and out. 

FIG. 10 is a diagiram illustrating yet another 
modulated graphics guide 1000 for augmented reality 
guided positioning of a needle, according to yet 
another illustrat IV© embodiment o£ the present 
invention. The modulated graphics guide 1000 
represents an extreme case of spatial modulation of the 

cylindrical graphics guide 500 o£ FIG, 9. Only a set 
of circles is depicted, distributed along and centered 
On the desired needle path 1050. In general, most o£ 
the graphics guides described herein may be rendered as 
sparse wireframes, particularly those graphics guides 
which are modulated according to the present invention. 
It is to be appreciated that temporally modulated 

guiding graphics advantageously have the added benefit 

of drawing attention to themselves, by differentiating 
themselves from an otherwise stationary scene. Thus, 



segments 93 O are ciat out of the cylindrical guide to 




temporally modulated guides are easy for a user to 
locate with respect to the background. 

FIG. 11A is a flow diagram illustrating a method 
for augmented reality guided instrument positioning 
using modulated graphics guides, according to an 
illustrative embodiment of the present invention. A 
graphics guide for positioning an instrument is 
determined (step 1102) . The graphics guide is rendered 
such that an appearance of at least one portion of the 
graphics guide is modulated with respect to at least 
one of space and time (step 1104) Step 1104 may 
include any one of steps 1104a, 1104b, and 1104c. 

With respect to step 1104a, the rendering step 
includes the step of varying a transparency of the at 
least one portion of the graphics guide with respect to 
other portions of the graphics guide to provide cl 
substantially unobstructed view through the at least 

OnG portion o£ the graphics guide to at least a portion 
of the instrument . 

With respect to step 1104b, the rendering step 

includes the step of varying a transparency of the at 

least one portion of the graphics guide during pre- 
defined time intervals to provide a substantially 
unobstructed view through the at least one portion of 
the graphics guide to at least a portion of the 
instrument during the pre-defined time intervals. 



With respect to step 1104c, the rendering step 
includes the step of sequentially varying a 
transparency of each of a plurality of portions (e.g., 
consecutive or non-consecutive portions) of the 
graphics guide during at least one pre-defined time 

interval to provide a substantially unobstructed view 

through each of the plurality of portions to at least 
port ion of the instrument as the transparency of each 
of the plurality of portions is varied during the at 
least one pre-defined time interval. 

PIG. 11B is a flow diagram illustrating another 
method for augmented reality guided instrument 
positioning using modulated graphics guides, according 

to another illustrative en^odiment of the present 

invention. A graphics guide for positioning an 
instrument is determined (step 1112) . Step 1112 
includes step 1112a, which constructs the graphics 

guide as a line or a cylinder. 

The graphics guide is rendered such that an 
appearance of at least one portion of the graphics 
guide is modulated with respect to at least one of 
space and time (step 1114) Step 1114 may include any 
one of steps 1114a, 1114b, and 1114c. 

with respect to step 1114a, the rendering step 

includes the step of modulating a transparency of the 
line/cylinder with respect to time so that the 
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line/cylinder repeatedly fades in and out of view to 
provide a substantially unobstructed view of the 
instrument when the line/cylinder is out of view. 

With respect to step 1114b, the rendering step 
includes the step of modulating a transparency of 
portions of the line/cylinder so that at least a 
portion of the instrument is substantially unobstructed 
when viewed through the portions of the line/cylinder. 

With respect to step 1114c, the rendering step 
includes the step of modulating a transparency of 
portions of the line/cylinder with respect to time and 
space so that at least a portion of the instrument is 
Substantially unobstructed when viewed through the 
portions of the line/cylinder during pre-defined time 
intervals . 

A description of graphics guides for use in a 

"bull's eye" or "down- the -beam" approach to augmented 
reality guided instrument positioning will now be given 
according to an illustrative embodiment of the 
invention . 

In the bull's eye method for augmented reality guided 
instrument placement, a user has an augmented view 
where the object and instrument are part of the real 
scene and a graphic target and/or a graphic guide are 
augmenting graphics. It is to be appreciated that the 
graphic target is registered with respect to the real 



target . The user assumes a viewpoint SQ that he QT She 
looks along the desired path towards the target. The 
user places the instrument so that it lines vip with the 
path. In the case of an elongated instrument such as a 
needle, correct alignment is achieved when the entire 

length o£ the instrument is seen as a dot (bull's eye) 
at the center of the target. Additional graphics for 
guidance (e.g., graphics path markers and instrument 
axis markers (hereinafter al SO referred bo as "axis 
markers") as described below) can be inserted in the 
augmented view to aid the user with the alignment. 

The "bull's eye" or Mown-the-beam" alignment 

method has been used in f luoro-guided medical 
procedures, e.g. f luoro-guided insertion of guide pins 

for pedicular bolts as described by Gennarelli et al., 

in "Minimally Invasive Techniques in Spinal Surgery: 
Current Practice", Neurosurg Focus 4(2): Article 8, 

1998. In a f luoro-guided procedure, the physician uses 

X-rays to check the alignment between instrument and 
target . In augmented reality guided instrument 

placement, the physician has "virtual X-ray vision" by 

seeing internal structures or markers superimposed on 

his real view of the patient. The superimposition is 

registered such, that a model of a tumor (or a 

corresponding marker) appears at the actual location of 
the tumor. Whereas in a f louro-guided procedure both 




the target and the instrument are observed as 
structures on the same X-ray image, the augmented 
reality guidance shows a graphical representation of 

the target (or relevant portion thereof) and a "real" 

(optical or video) image of the instrument. As noted 
above, additional graphics for guidance can be inserted 

in the augmented view to aid the user with the 

alignment . 

A brief description of some of the features and 

implementation considerations corresponding to the 

bull's eye or down- the -beam approach to augmented 
reality guided, instrument positioning will now be given 
according to an illustrative embodiment of the 
invention. In one embodiment of the invention, the 

approach is implemented using a 2 -step alignment 

process. First, the tip of the instrument is placed at 
the entry point. Then, the instrument is turned around 

it's tip to align the instrument with the desired path. 

Different graphics guides may be displayed for the 
different steps. Thus, in another embodiment, the user 
&e£S the target and an u alignable" instrument . An 
instrument is "alignable" when the user can look at the 
back end of the instrument and see that it lines up 

with his or her line of sight. That is, the user 

chooses a path by selecting a viewpoint when looking at 
the target, aligns the instrument with his or her line 



of sight towards the target, and inserts the instrument 
while keeping the alignment intact. 

The display of the target alone is sufficient to 
align an instrument with a path towards target, when 
the instrument is alignable. However, in other 
embodiments or the invention, guides (graphics path 
guides) marking a particular path may be used, as well 
as markers on the instrument (instrument axis markers 
Or aXlS markers) . In the other embodiments, the user 
sees path guides that mark a predetermined path and an 
axis marker on the instrument for aligning with the 
predetermined path via the path guides. The ascis 

markers may be part of the design of the instrument or 
may be added thereto. In the former case, the axis 

marker can either be a structure designed for the sole 

purpose of marking the instrument axis, or a structure 
designed to include the purpose of marking the 
instrument as well as having at least one other 
purpose, or the structure can be pre-existing and 
Simply used as an axis marker. 

It is to be appreciated that axis marker should be 
located at the end of the instrument (opposite from the 
front that is being inserted) that faces the viewer to 
facilitate alignment of the instrument with the path. 
Por example, the axis o£ the instrument maty be marked 

(axis marker) with a point or a circle that is to be 



aligned with a graphics guide. More sophisticated 
embodiments may include a. mechanical structure ( s ) that 
enables one to check alignment along an observer's line 
of sight. We call such an instrument "alignable" . For 
example , a ring may be placed around, the instrument's 
axis with a central pin sticking out along the axis. 
Other structures may be used, such as, for example, 
structures that are similar to aiming aids on guns. 
Thus, in general, an axis marker is marker at the back- 
end of an instrument that marks the instrument's axis. 
Preferably, the axis marker is protruding member. 
However, other structures may be used such as, for 
example, a circle or cross that is centered on the 
instrument's axis, and so forth. 

When the user wears a head-mounted display that 
gives him a variable viewpoint, a fixed path guide 

forces him to "stick" to that path and not stray off to 

a wrong instrument orientation due to an unnoticed 
shift in his viewpoint (in f louro-guided procedures, 
the X -F3.y flcuro machine's viewpoint is fixed during- 
the bull's eye instrument alignment). 

Preferably, graphic path guides are designed as 

(hollow) cylindrical structures. Rings around the axis 

(i.e. the desired path), or lines parallel to the axis 
may also be employed for- graphic path guides. However, 

it is to be appreciated that any shape or structure may 



be used to mark the path while maintaining the spirit 
and scope o£ the present invention. 

It is also preferable that the design of the 
graphics guides is adapted to the shape of the 
instrument. I£ an instrument is thicker (and 
unstructured) at the end that faces the viewer, then 
the guides should mark a circumference around the path 
large enough that the instrument can be seen inside the 
guiding structure . 

While any of the different types of graphics 

guides described herein may be combined, the bull's eye 

approach to augmented reality guided instrument 
positioning is amenable to the use of a depth marker to 
determine the correct depth o£ insertion o£ an 
instrument. A depth marker may be placed outside of 
the object, so that the correct depth of insertion is 
reached when a corresponding marker on the (external 
part) instrument lines up with the depth marker. 

PIG. 12 is a simplified diagram illustrating a 
needle 12 0 before and after being placed in alignment 
with an observer's line of sight, according to an 
ill ustrative embodiment o£ the present invention. 
Before the needle 1200 is aligned, i.e., when the 
needle 12 0 0 is not aligned with the observer's line of 

sight, the needle 1200 appears as a line 1200a. In 

contrast, after the needle 1200 is aligned with the 



observer's line of sight, the needle 1200 appears as a 
point or a dot 1200b. 

PIG. 12 is a diagram illustrating- a. conventional 
instrument 13 0 0 having a structure that renders the 
instrument 1300 "not alignable" . The instrument 1300 
is comprised of a thick handle 1310 with a thin 

elongated member 1320 protruding from an end thereof. 
The instrument 1300 is not alignable due to the big 
handle 1310 that hides the elongated member 1320 that 
is to be inserted into the object when viewed from the 

back of the instrument 1300. However, it is to be 

appreciated that an instrument may be designed to be 
al ignable . 

fig, 14 is a diagram illustrating various views of 

the instrument 1300 of FIG . 13, with a marker 1430 
located thereon according to an illustrative embodiment 

of the present invention. The marker is placed on the 

instrument by either the design of the instrument or 
modification of an existing design. The marker 143 0 is 

a protruding member that projects from the rear portion 

of the handle 1310 along the lengthwise axis of the 
instrument 1300. In this example, it is to be 

appreciated that the outer surface of the instrument 

1300, when viewed from the back, resembles a ring 1450. 
Moreover, a graphics marker 1470 such as a circle may 
be used to align the instrument such that a bull's eye 



is seen when the instrument 1300 is aligned, the bull's 
eye resulting from the protruding member 143 0 being 

centered in the graphics marker circle 14 7 0 which is 
also centered in the ring 1450 formed by the outer 

surface of the instrument 13 0 0 . 

FIG. 15 is a diagram illustrating graphics path 
guides 1510, 1520, and 1530 for indicating a path 1550 
for an instrument to follow to a target 1560, according 
to an illustrative embodiment of the present invention. 
In this example, the graphics path guides 1510, 1520, 

and 1530 are circles centered on and distributed along 

the path. Of course, other graphic structures may be 
used, while maintaining the spirit and scope of the 
invention. 

FIG. 16 is a diagram illustrating side and rear 
views of an instrument 1600 having an axis marker 1620 , 
according to an illustrative embodiment of the present 
invention. While the axis marker 162 0 is not visible 
in the side view, the axis marker 1620 is clearly 

visible in the rear view. In this case, the axis 
marker 162 0 is a circle or dot. 

FIGs. 17A-B are diagrams illustrating graphics 
path markers 1710, 1720, and 173 0 and an axis marker 
1750 for an instrument 1700, according to an 
illustrative embodiment of the present invention. In 
this case, the user first holds the instrument 1700 in 



a tilted fashion and aligns the tip 1705 of the 
instrument 1700 with an entry point on the object's 
surface 1703, guided by the path markers 1710, 1720, 
and 173 0 (FIG. 17A) . Then, the user swivels the 

instrument 1700 around its tip 1705 to align the back 

end of the instrument 1700 with the path markers (FIG. 

17B) . Thus, the instrument does not have to be 

"alignable" by itself. 

FIG. 18A is a flow diagram illustrating a method 
for augmented reality guided positioning of an 
instrument using a bull's eye alignment approach, 

according to an illustrative embodiment of the present 

invention. A viewpoint is established, from which a 
line of sight to a point on a target defines a path for 
an instrument to follow during a positioning of the 
instrument to the point on the target (step 1802) . 

From step 1802, it is to be appreciated, that the 
method of FIG. 18 may proceed to any of steps 1804, 
1806, and 1808. However, it is to be further 

appreciated that one o£ steps 1804 and 1806 may be 
skipped . 

At step 1804, the target may optionally be 

rendered as a graphics ob:ect. At step 1805, the path 

is marked with a plurality of graphical markers. At 
step 1808, the instrument is aligned along the line of 

sight to the point on the target. 




Step 1808 includes steps 1808a and 1808b. With 
respect to step 1808a, the tip of the instrument is 
positioned on the path, at an entry point on a surface 
of a physical object (e.g., patient) corresponding to 
the target. With respect to step 1808b, the instrument 

is rotated around the tip on the entry point until the 

instrument is aligned with the line of sight toward the 
point on the target . 

The instrument is moved along the path towards the 
point on the target, subsequent to the aligning step(s) 
(step 1810) . 

FIG. 18B is a flow diagram illustrating another 
method for augmented reality guided positioning of an 
instrument, according to another illustrative 
embodiment of the present invention. At least one 
graphics path guide is rendered for indicating a path 

for an instrument to follow to a target (step 1842) . 

At least one axis marker on the instrument is 

identified that marks am axis of the instrument (step 
1844) . The axis is disposed from the front portion to 
the back portion of the instrument, the front portion 

corresponding to the point on the target, and the back 

portion facing a user. 

From step 1844, the method of FIG. 18B may 
continue to any one of steps 1846, 1848, 1850, and 
1852 . 



With, respect to step 1846, trie at least one a^is 

marker is constructed and rendered as a graphical 
marker, and the instrument is tracked to register the 
cfnraphica.1 marker with respect to the axis of the 

5 instrument. 

With respect to step 1848, an existing feature of 
the instrument is selected to be the at least one axis 

marker , 

With respect to step 1850, the instrument is 
10 designed to include the at least one axis marker. 

With respect to step 1852, the at least one axis 

marker is added to a structure of the instrument, 

The at least one axis marker is then aligned with 
respect to the at least one graphics guide to align the 
15 instrument to the path (step 1854) . 

With respect to line-of -sight guided positioning 

of an instrument, a description of display options 

corresponding thereto will now be given according o an 
illustrative embodiment of the present invention. The 
20 user may wear a head-mounted display of the optical - 
see-through variety, or of the video -see -through 

variety with head-mounted camera. The video camera (or 
video cameras) can also be externally mounted in a way 
that it can be aligned with a chosen path and fixed in 

25 that pose, with the augmented view displayed on a 
monitor . 




A description of virtual line-of -sight guided 

positioning of an instrument according to an 
illustrative embodiment of the present invention will 
now be given . In this approach , the instrument is 

5 tracked. The pose of the instrument is known and is in 
the same coordinate system in which we know the 
location of the target. A graphical representation of 
both the target and the instrument can be rendered from 
an arbitrary virtual viewpoint . 

10 To perform an instrument-to- target alignment, we 

can be guided completely by this virtual scene that 
shows the geometric relationship between the locations 
of the target and the instrument. A virtual bull's eye 
view is very helpful and intuitive to perform the 
15 alignment . 

Before, the user was provided with an augmented 
view o£ the real scene so that the chosen path of 
instrument positioning was along the line-of -sight 
towards the target . 

20 Now, the user is being provided with a virtual view for 

which the line-of -sight towards the target is aligned 
With a Chosen path o£ the instrument positioning. The 
virtual view shows a graphical representation of the 
target ("virtual target") and a graphical 

25 representation of the instrument ("virtual 

instrument " ) . 



FIG. 19 is a diagram illustrating a graphics 
instrument guide (virtual instrument) 1910 and 192 0 and 
a graphics target guide (virtual target) 1940 for 
virtual 1 ine -of - sight guided positioning of an 
instrument according to an illustrative embodiment of 
the present invention. A point on the actual target is 
rendered as a graphics target guide 1940 illustrated as 
a disc, and the instrument is marked by a graphics 
instrument guides 1910 and 1920 consisting of two rings 
centered on the instrument's axis 1960 and located at 

different positions along the instrument's axis 1950. 

Two different views are shown in FIG. 19, one for the 
case when the graphics instrument guide 1910 and 1920 

is not aligned with the graphics target guide 1940 and 

the other case when the graphics instrument guide 194 0 
is aligned with the graphics target guide 1940 . 

FIG. 2 0 is a diagram illustrating another graphics 
instrument guide 2 010 and 2 02 0 and another graphics 

target guide 2040 for virtual line-of-eight guided 

positioning of an instrument according to another 
illustrative embodiment of the present invention. The 

actual target is marked with a graphics target guide 

2040 consisting of a ring, and the instrument is marked 
with graphics instrument guides 2 010 and 2 02 0 
consisting of a disc and a ring, both centered on the 
instrument's axis 2060 and located at different 




positions along instrument's axis 2 060. Two different 
views are shown in FIG. 20, one for the case when the 

graphics instrument guide 2010 ana 2020 is not aligned 

with the graphics target guide 2 04 0 and the other case 
5 when the graphics instrument guide 2 04 0 is aligned with 

the graphics target guide 2040. 

With respect to virtual line-of -sight guided 
positioning of an instrument, a description of adaptive 

rotation of a virtual view around a path will now be 

10 give according to an illustrative embodiment of the 

present Invention. To make alignment of the instrument 
(guided by the virtual view along the path) most 
intuitive, the virtual view should be presented to the 
user m an orientation such that i£ the user- moves the 

15 rear end of the instrument to the left, the graphics 
representation of the instrument should show the 
instrument's rear end moving to the left. The same is 
true for the cases of moving right, forward, backward, 

and so forth. If the user's position is variable, the 

20 system may adapt the orientation of the virtual down- 
the-path view. 

With respect to virtual line-of -sight guided 

positioning of an instrument, a description of the 
combination of a side view and a bull's eye view will 
25 now be given according to an illustrative embodiment of 
the present invention. Besides a virtual down- the -path 
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view, the user may also be provided with a side view of 

the scene, virtual or augmented. The side view helps to 

control the movement of the instrument along its way 
towards the target (depth of instrument insertion) . 

5 With respect to virtual line-of-sight guided 

positioning of an instrument, a description of the 

integration of depth information into a virtual along 

the path view will be given. Information about the 
forward movement of the instrument along the path 

10 towards the target can be integrated into the virtual 

down- the -path view. 

FIG. 21 is a diagram illustrating graphic depth 
markers for graphically representing instrument depth 
in virtual line-of-sight guided positioning of an 
15 instrument, according to an illustrative embodiment of 

the present invention. Instrument depth corresponds to 
the relationship between graphic instrument depth 

marker 2110 and graphic target depth marker comprised 
of 2150 and 2160. Graphic instrument depth marker and 
20 graphic target depth marker move closer to each other 

as the instrument moves towards the target . When 

graphic instrument depth marker and graphic target 

depth marker line up as shown in the FIG. 21, target 
depth has been reached. 
25 FIG. 22 is a diagram illustrating graphic 

instrument markers (virtual instrument) and. graphic 



target markers (virtual target) that simultaneously 
allow one to judge alignment with respect to the path 
and with respect to the depth, according to another 
illustrative embodiment of the present invention. The 
5 graphical target markers consist of four concentric 
circles 2250, 2260, 2270, 2280, their common center 
corresponding to the target point. The graphic 

instrument markers consist o£ two circles 2210 and 
2220, centered on an axis 2240 corresponding to the 
10 instrument's axis, and located at different locations 

along the axis. When the path is aligned correctly, 

the graphical target markers and the graphical 
instrument markers appear concentric. Also, if the 

diameters of the markers appear so that each instrument 

15 circle falls between two corresponding target circles , 
the correct depth is reached. a.s well. 

FIG. 23 is a flow diagram illustrating a method 
for virtual line-of -sight guided positioning of an 
instrument, according to an illustrative embodiment o£ 

20 the present invention. A point is defined on an actual 
target and an actual path to reach the point on the 
actual target (step 2302) . A pose of an actual 

instrument is tracked with respect to a pose of the 
actual target (step 2304) . 

25 An orientation is chosen of the graphical 

representation around the virtual line of sight 
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according to a pose of a user with respect to the 

actual target (step 230S) . Step 2305 may include steps 

2306a, 2306b, and 2306c. With respect to step 2306a, 
the orientation is determined such that east, west, 

5 north, and south correspond to right, left, forward, 

and backward, respectively, for the pose of the user in 
which the user- faces the actual target . The 
determining step (2306a) is based on a selection. With 
respect to step 23 06b, the orientation is dynamically 

10 adjusted according to a change of the pose o£ the user. 

With respect to step 2306c, the selection is 
dynamically adjusted with respect to the pose of the 
user . 

A graphical representation is rendered of the 
15 actual instrument and the actual target point to obtain 

a virtual instrument and a virtual target point, 
respectively (step 2308) . The graphical representation 

is rendered with respect to a virtual viewpoint from 

which a virtual line of sight to the virtual target 
20 point coincides with a virtual path for the virtual 

instrument to follow during a positioning of the actual 
instrument to the point on the actual target . The 
virtual path corresponds to the actual path. 

Graphical information is rendered about a distance 
25 between the actual instrument and the point on the 
actual target (step 2310) . The rendering step may 




performed according to a virtual camera with a wide 
angle lens. The graphical information about the 
distance is overlaid onto the graphical representation 
(step 2312) . 

5 The graphical representation from the virtual 

viewpoint is combined with another graphical 
representation from another virtual viewpoint looking 
at the virtual path from the side (step 2314) . The 
graphical representation from the virtual viewpoint is 

10 Combined with an augmented reality view (step 2315) . 
It is to be appreciated that step 2314 and 2316 are 

optional and one, both, or none may be performed in 

various implementations of the method of FIG. 23. 

The virtual instrument is aligned along the 

15 virtual line of sight to the virtual tar-get point to 

align the actual instrument along the actual path (step 
2318) . The actual instrument is moved along the actual 

path towards the point on the actual target (step 

2320) . 

20 A description of graphics proximity markers for 

augmented reality guided instrument positioning will 
now be given according to an illustrative embodiment of 
the present invention. 

A graphics proximity marker is a virtual depth 

25 gauge that is placed into an augmented view of an 

object into which the user wants to insert an 




instrument. The user inserts the instrument until a 
(real) marker on the instrument reaches the graphics 
proximity marker. At this point the insertion is 

complete, the tip of the instrument has reached the 

5 target . 

The distance between the target and. the graphics 

proximity marker is the same as the distance between 
the tip o£ the instrument and a marker on the 
instrument ("instrument marker"). The user inserts the 

10 instrument along a path that may also be marked with 

virtual guides until the instrument marker reaches the 
graphics proximity marker . Now the nser knows that the 

tip of the instrument has reached the target even 
though he does not directly see the instrument's tip. 
15 It is to be appreciated that more than one 

graphics proximity marker may be used to position an 
instrument. For example, a first graphics proximity 
marker may be used to indicate the outer surface of a 
target volume (e.g., a mass for which a biopsy is 

20 desired) , and a second graphics proximity marker may be 

used to indicate an inner portion of the target volume 

(e.g., the center of the mass) at which the needle may 

be actuated by the user to perform the biopsy. It is 

to be further appreciated that two or more graphics 

25 proximity markers may be used to indicate a 

predetermined range (e.g., a range in which the needle 




is within the mass and not external with respect 
thereto) . 

Consider the task of inserting an instrument (e.g. 
a needle) into an object (e.g. a patient) where the 

5 instrument has to reach a predetermined target. The 

user does not have a direct view of the target, but the 
presence and position of the target as an internal 

structure of the "object" has been predetermined with 

some imaging means (e.g. CT scan) . 

10 Further consider that trie user is guided to 

perform his task by augmented reality visualization. 
Superimposed on his or her view of the object, the user 
sees a graphical representation of the target, 
correctly registered with the actual position of the 

15 target. This augmented Vl£w is preferably an stereo to 

give the user 3D perception. Besides the target, 

graphical guides can be added ho the augmented view to 
support the user in the correct insertion of the 
instrument . 

20 If we do not track the instrument, we cannot 

visualize the hidden (inserted) parts of the 
instrument . We cannot see the tip of the instrument as 
it reaches the target. We only see the object and the 
part of the instrument outside of the object as well as 

25 the graphical information (target and/or guides) that 

we choose to overlay onto the view. 




To find the right path, we align the outside part of 

the instrument (the part that is not yet inserted and 
hence visible to the direct view) with graphical 
(virtual) guides that we positioned in the augmented 
5 view to mark the path outside of the object. To find 
the right depth, we align a marker on the outside part 
of the instrument with a graphical proximity marker 
(virtual depth gauge) that we positioned outside of he 
object to mark the proximity of the instrument marker 

10 for the final desired instrument position. It is to be 
appreciated that virtual depth gauges can be applied 
for methods where the user looks at the path from an 
oblique angle as well as when the user looks along the 
path ("down-the-beam") as described herein. 

15 FIG. 24 is a diagram illustrating a graphics depth 

marker 24 0 0 for augmented reality guided positioning of 
an instrument , according to an illustrative embodiment 
of the present invention. In this example, the exact 
instrument path is not critical. The depth marker 2400 

20 is implemented as a mesh that marks a predetermined 

distance from the target 2420, i.e., the mesh lies on 

part of a spherical surface with the target 2420 at trie 

center . 

FIG. 25 is a diagram illustrating a graphics depth 
25 marker 2500 added to a graphics path guide 2510 for 

augmented reality guided positioning of an instrument, 



according to an illustrative embodiment of the present 

invention. In this example , the graphics path guide 
2510 is normal to the desired path 2520 and is 
implemented as a slit. Of course, instead of a slit, 
any other kind of graphical marking that can mark a 
position along the path can be used, e.g., a darker 

line, an arrow, and so forth. The depth marker need 

not show any symmetry. For example, the depth marker 
can foe planar, can surround the path in a ring- like 

structure, or can take be embodied as another 3D shape. 

FIG. 2 6 is a diagram illustrating instrument 
markers used as depth markers for augmented, reality 
guided positioning of an instrument, according to an 
illustrative embodiment of the present invention. It 

is to be appreciated that an existing feature of an 

instrument can be used as a depth marker (like 2610) , 
as well as an added feature. The features can be 
surface structures (like 2620) or protruding or 
indented structures (like 2630 and 2640, respectively) , 

or any other structure that can be used as a mark. In 

particular, they need not be rings as in the 
illustrated examples. 

FIG. 2 7 is a flow diagram illustrating a method 
for indicating instrument depth for augmented reality 
guided instrument positioning, according to an 
illustrative embodiment of the present invention. At 
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least on© graphics proximity marker is determined for 
indicating a proximity of a predetermined portion of an 
instrument to a target (step 2702) . The proximity may 
be based on, for example, a final forward position of 
5 the predetermined portion of the instrument with 

respect to the target, a first measure of proximity for 
indicating an outer surface of a target volume and a 
second measure of proximity for indicating an inner 
portion o£ the target volume, a first measure of 

10 proximity for indicating a front portion of a target 

volume and a second, measure of proximity for indicating 
a back portion of the target volume (the front portion 
corresponding to entering the target volume and the 
back portion corresponding to exiting the target 

15 volume) . 

At least one graphics path marker is determined 

for identifying at least one path for the instrument to 
the target (step 2704) . 

The at least one graphics proximity marker is 

20 rendered such that the proximity of the predetermined 
Portion o£ the instrument to the target is 
ascertainable based on a position of a marker on the 
instrument with respect to the at least one graphics 

proximity marker (step 2706) . The at least one 

25 graphics path marker- is rendered such that the at least 




one path is identified by the at least one graphics 
path marker (step 2708) . 

In general, it is to be appreciated that monoscopic 

vision is preferable to stereoscopic vision for the 
5 bull's eye alignment approach described herein, and 

stereoscopic vision is preferable to monoscopic vision 
for depth, alignment (unless the user is willing to move 
his or her head and look from the side for depth 
checks) . For* monoscopic vision, the "user may j ust close 

10 one eye, or the system can be switched between 

monoscopic and. stereoscopic vision (for monoscopic 
vision the same video image is displayed on both 
screens of the stereo head-mounted display), or bull's 

eye alignment in s teres cop ic vision is made easier h>y 

15 showing the real scene in stereo, but overlaying the 
graphic guides only onto one of the views (preferably 

the one for the user's dominant eye) . 

Although the illustrative embodiments have been 

described herein with reference to the accompanying 

20 drawings, it is to be understood that the present 

invention is not limited to those precise embodiments, 
and that various other changes and modifications may be 

affected therein by one o£ ordinary skill in the 
related art without departing from the scope or spirit 
25 of the invention. All such changes and modifications 
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are intended to be included within the scope of the 
invention as defined by the appended claims. 



